Method to imitate lifelike images for computer deformed objects

ABSTRACT

A method to imitate lifelike images for computer deformed objects includes the following steps, selecting a number of significant points on an object, categorizing the significant points into measurable points and specific points, defining a relationship between the measurable points and the specific points with a statistical regression model, creating a table of weighted values for the specific points and animating an object based on the weighted value table. The present invention uses the measured displacement of a number of measurable points to calculate expected positions of the specific points to reduce processing time and create real time images. The method uses a statistical analysis method to decrease access time to a database, save storage space of the database and create lifelike, real-time and animated images in accordance with the object as the object changed its surface.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method for creating lifelike imagesfor computer deformed objects, and more particularly to a method toimitate lifelike images, which uses a statistical regression analysismethod to create a weighted value table that is used to estimatepositions of desired specific points on the deformed object or roboticface.

2. Description of Related Art

Computer deformed objects and animated facial expressions of robots needto be lifelike so that the deformed objects look real. Methods forimitating and creating lifelike images for computer deformed objects inaccordance with prior art can be found in U.S. patents by Kelly Dillon,U.S. Pat. No. 6,020,892, Christian Rouet etc, U.S. Pat. No. 5,818,461,Pierre LaChapelle, U.S. Pat. No. 6,163,322, Erich Haratsch, et. al.,U.S. Pat. No. 6,483,513 B1, Eric Cosatto, et. al.,. U.S. Pat. No.5,995,119, Tsutomu Ando, U.S. Pat. No. 6,476,815 B1 and Volker Blanz,et. al., U.S. Pat. No. 6,556,196 B1.

The conventional methods for creating lifelike images in theaforementioned patent documents typically use either interpolation orimage composition methods to imitate animated images and can be dividedinto the following categories.

The first conventional method prepares many images of different shapesor expressions by deforming a physical object before animating anobject. The prepared images are stored in a database, and the imageswill occupy a huge storage space in the database. Therefore, the firstmethod to animate an object requires observation of changes in the shapeof a similar physical object and identifying stored images for an imagethat that are close to a recent image of the physical object as thephysical object becomes deformed.

The second conventional method uses interpolation to calculate positionsof some desired points on imitated lifelike images to create theanimated images of the physical object. The images of the physicalobject must be observed before and after the physical object changes,and then using an interpolation method, the desired points on theanimated images are calculated. Such a method is included in the 3DS MAXsoftware uses a morphing function to embody the animated images.

The third conventional method is similar to the first method, but thethird method uses a method of calculating proportional differencesbetween the different representative images to form lifelike compositesof the animated images of the deformed objects.

The fourth conventional method determines several measurable points andspecific points on the physical object's surface, and then establishes aweighted value table by calculating distance differences between each ofthe specific points to all the measurable points. When the physicalobject's appearance or shape has changed, the displacements of all themeasurable points are measured, and the positions of the specific pointscan be found by a relationship related to the displacement of themeasured measurable points and the weighted value table. Thereafter, thedeformations can be created since the positions of all the measurablepoints and the specific points are known after the physical object haschanged shape. This method is included in the MAYA and 3DS MAX software.

However, the first and the second conventional methods require a hugedatabase to store the pre-processed images and can only imitate an imagethat was created and stored in the database prior to the animation. If adesired image cannot be found in the database, the desired animatedimage cannot be created. Furthermore, the time required to search theentire database for a stored image with the two methods makes real-timeanimation prohibitive.

The third conventional method also requires a huge database to storerepresentative images for image composition. The application of thethird conventional method is limited by the quantity of representativeimages stored in the database. Although, an improvement called PrincipalComponent Analysis (PCA) has been incorporated in the third conventionalmethod to reduce the storage space required in the database for therepresentative images. However, PCA necessitates a great deal ofpre-processing, which takes time.

The fourth conventional method requires a specialist, such as aprofessional animator, to create the weighted value table based on thephysical features of the object, such as muscle distribution of a body.Creating the weighted value table is a huge task and is particularlydifficult if the physical features of the object cannot be clearlydistinguished or found.

To overcome the shortcomings, the present invention provides a method toimitate living images in real-time for computer deformation of objectsin real-time to mitigate or obviate the aforementioned problems.

SUMMARY OF THE INVENTION

The main objective of the present invention is to provide a method toimitate living images in real-time for computer deformation of objects.

Another objective of the present invention is to provide a method toimitate lifelike images that uses a statistical model to reducecalculations in the creation of data for a weighted value table andprovide precise data to perform an animation.

The method to imitate lifelike images for computer deformed objectscomprises the steps of selecting a number of significant points on anobject such as human face, categorizing the significant points intomeasurable points and specific points, defining a relationship betweenthe measurable points and the specific points with a statisticalregression analysis, creating a table of weighted values for thespecific points and either animating an object based on the weightedvalue table.

The method will save a lot of time in pre-processing the object toestablish a weighted value table that contains all the weighted values.In addition, the statistical analysis method is a precise analysismethod so that the estimated positions of the specific points are veryclose to the exact positions of the specific points on the real object.

Since the method only demands to store relevant weighted values, storagespace for the weighted values in a database is smaller than the storagespace required in cited prior art methods. Also, the calculation for thedeformation images is not complex such that the simulation can berapidly performed to create real-time images.

Other objectives, advantages and novel features of the invention willbecome more apparent from the following detailed description when takenin conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a photograph of a person's face with multiple significantpoints marked in accordance with the present invention;

FIG. 2 is a front plan view of animated image with all the significantpoints in FIG. 1 refer to FAP of MPEG 4 format;

FIG. 3 is a series of base expressions in accordance with the presentinvention used to create a weighted value table;

FIG. 4 is a photograph of a person's face with the measurable points inFIG. 1 changing from a neutral expression to a later expression;

FIG. 5 a photograph of a person's face in FIG. 4 with simulatedpositions of the specific points;

FIG. 6 is an operational front plan view of an animated mouth of theface in FIG. 4 with the displacements of the measurable points when themouth changes its shape; and

FIG. 7 is a flowchart of the method in accordance with the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The following detailed description of a method to imitate lifelikeimages for computer animated objects in accordance with the presentinvention specifically describes the method to imitate lifelike imagesfor a computer animated face. The method can be used to imitate lifelikeimages for control of robotic facial expressions, which is the same andis not further described.

With reference to FIGS. 1 and 7, the method in accordance with thepresent invention provides a solution to simulate and find predictedpositions of multiple specific points (102) on a surface of an object,such as a face (10) after the object changed its shape. The positions ofthe specific points (102) are related to the simulation of the changedshape of the face (10) and are predicted by measuring and analyzingpositions of multiple measurable points (101) on the face (10). If theestimated positions of those significant points including the measurableand the specific points (101, 102) are very close to the real positionson the changed shape of the face (10), the simulation for the facialimages will become living and lifelike. The method in accordance withthe present invention uses a statistic analysis method of a regressionrelationship, such as first-order or multi-order multiple linearregression models to deal with the positional relationships of thesignificant points (101, 102). The positions of the all significantpoints (101, 102) are described by coordinates that can be eitherthree-dimensional or two-dimensional.

With reference to FIGS. 1 and 2, the step of selecting a number ofsignificant points on a surface of an object may be implemented toselect multiple significant points on a face (10) that shows a neutralexpression. The selected significant points are selected at positionsthat change measurably and distinctly as the object changes shape andserve as reference points to reflect changes in the shape of an object.

The step of categorizing the significant points comprises selecting anumber of measurable points (101) and a number of specific points (102).The measurable points (101) are at positions that allow theirdisplacements to be definitively observed and measured easily before andafter the face (10) changes its expression. For example, the measurablepoints (101) are located respectively alongside the outer edge of themouth on the face (10) and at the centers of the eyeballs, because thechanges of those positions can be clearly observed before and after theface (10) changes its expression. The measurable points (101) are markedrespectively with white crosses and are indicated individually withreference numbers A₁ to A₁₀. For subsequent calculations, K is thenumber of measurable points (101).

Likewise, the specific points (102) are at positions that can bemeasured when the face is neutral but need to be determined after theface (10) changes its expression so the features can be renderedappropriately. In the example, the specific points (102) are locatedrespectively alongside the edge of the face (10), on the cheeks andedges of the nose. The simulated images can be precisely rendered withrespect to the real face (10) if the measured and estimated positions ofall the measurable points (101) and the specific points (102) are known.The specific points (102) are marked with white dots in FIG. 1 and areindicated individually with reference numbers F₁ to F₂₅ in FIG. 2. Forsubsequent calculations, L is the number of specific points (102).

The step of defining a relationship between the measurable points (101)and the specific points (102) with a statistical regression modelcomprises the steps of changing the shape of the face (10) to create aseries of base expressions, measuring the displacements of themeasurable points (101) and the specific points (102) for each baseexpression and using a statistical regression model to define arelationship between the measurable points (101) and the specific points(102).

With reference to FIG. 3, the shape of the face (10) is changed tocreate a series of base expressions (13), and m is the number of baseexpressions (13), where m≧(K+1). The number of the base expressions (13)is greater than the number of the measurable points (101) means that ifthe more base expressions (13) are selected, the computer images will bemore lifelike. Only a small number of the base expressions (13) areshown in FIG. 3.

Measuring the displacements of all the significant points (101, 102) foreach base expression comprises measuring the displacement of eachsignificant point (101, 102) from its initial position in the neutralexpression to its position in each one of the base expressions (13).Therefore, the displacements of the all significant points (101, 102) isknown for each of the base expressions (13).

Defining a relationship between the measured displacements of themeasurable points (101) and the specific points (102) is establishedwith a statistical regression model based on the measured displacementsof the significant points (101, 102).

The statistical regression model may be a first-order or multi-ordermultiple linear regression model.

For example, a first-order multiple linear regression model has beenestablished as:$F_{i} = {{\sum\limits_{n = 1}^{K}{A_{n}W_{ni}}} + C_{i}}$(Statistic Multiple Regression Model)

where

F_(i) is denoted as the displacement of the ith specific point (102),

A_(n) is denoted as the displacement of the nth measurable point (101),

W_(ni) is denoted as a weighted value corresponding to the ith specificpoint (102) and the nth measurable point (101), and

C_(i) is a constant value.

The displacements of the nth measurable point (101) for each of the baseexpressions (13) are respectively denotes as a_(1n), a_(2n), . . . ,a_(mn). Likewise, the displacements of the ith specific point (101) foreach of the base expressions (13) are respectively denoted as f_(1i),f_(2i), . . . , f_(mi).

Consequently, the first-order multiple linear regression model can berewrite in matrix form as follow: $F_{i} = {\begin{bmatrix}f_{1i} \\f_{2i} \\\vdots \\f_{m\quad i}\end{bmatrix} = {{A\quad W_{i}} = {\begin{bmatrix}a_{11} & \cdots & a_{1K} & 1 \\a_{21} & ⋰ & a_{2K} & 1 \\\vdots & \vdots & ⋰ & \vdots \\a_{m1} & \cdots & a_{mK} & 1\end{bmatrix}\begin{bmatrix}w_{1i} \\\vdots \\w_{Ki} \\c_{i}\end{bmatrix}}}}$

The step of creating a table of weighted values for the specific points(102) is implemented to solve the aforesaid matrix form with astatistical solution of least square approximation (LSA) to calculatethe variable elements of the weighted value matrices W_(1i) to W_(Ki)denoted as a combined weighted value matrix W_(i)*. The combinedweighted value matrix W_(i)* can be calculated with matrix calculationas follows:

A^(T)F_(i)=A^(T)AW_(i), where A^(T) is a transpose of matrix A,

W_(i)*=(A^(T)A)⁻¹A^(T)F_(i), where (A^(T)A)⁻¹ is an inverse of matrixA^(T)A.

Therefore, a weighted value table can be established with respect toeach of the combined weighted value matrix W_(i)*, which is listed asfollows.

Calculated Weighted Value Table

Calculated Weighted Value Table F₁ . . . F_(i) . . . F_(L) A₁ W₁₁ . . .W_(1i) . . . W_(1L) . . . . . . . . . . . . . . . . . . A_(n) W_(n1) . .. W_(ni) . . . W_(nL) . . . . . . . . . . . . . . . . . . A_(K) W_(K1) .. . W_(Ki) . . . W_(KL) C C₁ . . . C_(i) . . . C_(L)

The step of either animating an object based on the table of weightedvalues comprises the following steps: changing the shape of the face(10), measuring the displacement of the measurable positions (101),calculating expected positions of the specific points (102) and usingthe measured and calculated positions respectively of the measurablepoints (101) and the specific points (102) to generate an animatedobject. The weighted values are employed to weight the displacement ofthe measurable points (102) when calculating the estimated positions ofthe specific points (102) so as to perform a simulation to createlifelike images about the face (10).

When the weighted value table has been created, the deformation can beperformed with the displacements of a number of measurable points (101).Because the variable elements of the displacement matrix A of themeasurable points (101) is known after the facial expression is changed,the displacements of each of the specific points (102) can be simulatedby a calculation of the aforesaid multiple linear regression model,$F_{i} = {{\sum\limits_{n = 1}^{K}{A_{n}W_{ni}}} + {C_{i}.}}$The animated images of the face (10) can be simulated to createlife-like computer animations or to control the facial expressions of arobot when the displacements of all points (101, 102) are known.

Therefore, the animated object can be established by tracking thechanges of the number of measurable points (101) as the face (10)changes its shape, calculating simultaneous positions of the specificpoints (102) with the corresponding weighted values and generating theanimated images. Since the positions of only a number of measurablepoints (101) need to be measured, the images created by the presentmethod will be animated in real time.

With reference to FIGS. 4 to 6, to further describe the step of eitheranimating an object or controlling a robot's facial expressions of themethod of the present invention, a further example is provided toexplain the effects of the present invention. A neutral expression (11)of the face (10) is observed and shown at the left side of FIG. 4, andthe measurable points (101) and desired specific points (102) aredetermined on the facial surface as previously described after theweighted value table has been created. Then the neutral expression (11)is changed to a later expression (12) shown at the right side of FIG. 4,such as opening the mouth (103) from the rest, which is alsoschematically illustrated in FIG. 6. The positions of the measurablepoints (101) and the specific points (102) have been changed from theirinitial positions. When the displacements of the measurable points (101)are measured, the positions of the specific points (102) can besimulated by the aforesaid multiple linear regression model incooperation with the created weighted value table to calculate theirpositions. The simulated positions of the specific points (102) areshown on the face (10) of the later expression (12) with white dots asillustrated in FIG. 5. The simulated positions of the specific points(102) shown in FIG. 5 are almost the exact positions with respect to thereal positions of the specific points (102) of the changed neutralexpression (11).

Consequently, the present invention can be used to precisely simulatelifelike images of shapes or surfaces of an object by tracking themeasurable points (101) as the object changes its shape or surface.Meanwhile, the positions of the specific points (102) can be calculatedwith the weighted values to create real time and lifelike images. Thepositions of all significant points (101, 102) can be used to createcomputer animations or control to present robotic facial expressionswith respected to the physical object.

Even though numerous characteristics and advantages of the presentinvention have been set forth in the foregoing description, togetherwith details of the structure and function of the invention, thedisclosure is illustrative only, and changes may be made in detail,especially in matters of shape, size, and arrangement of parts withinthe scope of the appended claims.

1. A method to imitate lifelike images for computer deformed objects,and the method comprising: (1) selecting a number of significant pointson a surface of an object; (2) categorizing the significant points intoa number of measurable points and a number of specific points; (3)defining a relationship between the measurable points and the specificpoints with a statistic regression analysis by a. changing the surfaceof the object to create a series of base deformations; b. measuringdisplacements of all the measurable points and the specific points foreach of the base deformations; and c. using the statistic regressionanalysis to define the relationship by analyzing the measured positionsof all the measurable points and the specific points for each baseexpression; (4) creating a table of weighted values for the specificpoints based on solving the statistic regression analysis; and (5)animating a deformed object with following steps: a. changing thesurface of the object; b. measuring displacements of the measurablepoints; c. calculating predict positions of the specific points with themeasured displacements of the measurable points and the table ofweighted values; and d. using measured positions of all the measurablepoints and the calculated positions of all the specific points to createthe deformed object.
 2. The method to imitate lifelike images forcomputer deformed objects as claimed in claim 1, wherein the statisticregression model is a first-order multiple linear regression model, andthe first-order multiple linear regression model is solved by a leastsquare approximation to calculate the weighted values.
 3. The method toimitate lifelike images for computer deformed objects as claimed inclaim 1, wherein the statistic regression model is a multi-ordermultiple linear regression model, and the multi-order multiple linearregression model is solved by a least square approximation to calculatethe weighted value.
 4. The method to imitate lifelike images forcomputer deformed objects as claimed in claim 1, wherein the surface ofthe object is two-dimensional whereby the displacements of all themeasurable points and the specific points for each of the baseexpressions are measured from the changed surface of the object.
 5. Themethod to imitate lifelike images for computer deformed objects asclaimed in claim 2, wherein the surface of the object is two-dimensionalwhereby the displacements of all the measurable points and the specificpoints for each of the base expressions are measured from the changedsurface of the object.
 6. The method to imitate lifelike images forcomputer deformed objects as claimed in claim 3, wherein the surface ofthe object is two-dimensional whereby the displacements of all themeasurable points and the specific points for each of the baseexpressions are measured from the changed surface of the object.
 7. Themethod to imitate lifelike images for computer deformed objects asclaimed in claim 1, wherein the surface of the object isthree-dimensional whereby the displacements of all the measurable pointsand the specific points for each of the base expressions are measuredfrom the changed surface of the object.
 8. The method to imitatelifelike images for computer deformed objects as claimed in claim 2,wherein the surface of the object is three-dimensional whereby thedisplacements of all the measurable points and the specific points foreach of the base expressions are measured from the changed surface ofthe object.
 9. The method to imitate lifelike images for computerdeformed objects as claimed in claim 3, wherein the surface of theobject is three-dimensional whereby the displacements of all themeasurable points and the specific points for each of the baseexpressions are measured from the changed surface of the object.
 10. Themethod to imitate lifelike images for computer deformed objects asclaimed in claim 7, wherein the object is a human face.